Using augmented reality for real-time visualization of tactile health examination

An augmented-reality approach to real-time visualization of tactile data with projection on a human organ is presented. A target procedure is breast cancer examination with a tactile sensor. The special tactile sensor is used to measure tissue stiffness values at discrete points of the examined organ. We developed an augmented reality system which integrates the tactile sensor, a head-mounted display with a small video camera, and a notebook computer. Tracking of the human organ and the tactile sensor is based on optical markers. One marker is attached to the patient’s body and is used to track the spatial position of the human organ. Another marker is placed on the tactile sensor. Registering space positions of this marker with respect to the first marker allows determination of the shape of the human organ for subsequent data visualization. Results of stiffness measurements are depicted as semi-transparent three-dimensional objects projected on the patient’s body. Different visualization techniques are employed depending on the amount of data and user preference. Experiments with sensing tactile data and its visualization for a variety of objects, including a silicon breast model and animals organs, have been performed. It was found that the developed augmented-reality system is useful in medical-physics measurements.